Arbitrary visual features as fiducial elements

ABSTRACT

Systems and methods for registering arbitrary visual features for use as fiducial elements are disclosed. An example method includes aligning a geometric reference object and a visual feature and capturing an image of the reference object and feature. The method also includes identifying, in the image of the object and the visual feature, a set of at least four non-colinear feature points in the visual feature. The method also includes deriving, from the image, a coordinate system using the geometric object. The method also comprises providing a set of measures to each of the points in the set of at least four non-colinear feature points using the coordinate system. The measures can then be saved in a memory to represent the registered visual feature and serve as the basis for using the registered visual feature as a fiducial element.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/412,784, filed on May 15, 2019 and entitled “ARBITRARY VISUALFEATURES AS FIDUCIAL ELEMENTS”, which is incorporated herein byreference in its entirety.

BACKGROUND

Fiducials elements are physical elements placed in the field of view ofan imager for purposes of being used as a reference. Geometricinformation can be derived from images captured by the imager in whichthe fiducials are present. The fiducials can be rigidly attached to theimager itself such that they are always within the field of view of theimager or placed in a locale so that they are in the field of view ofthe imager when it is in certain positions within that locale. In thelater case, multiple fiducials can be distributed throughout the localeso that fiducials can be within the field of view of the imager as itsfield of view is swept through the locale. The fiducials can be visibleto the naked eye or designed to only be detected by a specializedsensor. Fiducial elements can be simple markings such as strips of tapeor specialized markings with encoded information. Examples of fiducialtags with encoded information include AprilTags, QR Barcodes, Aztec,MaxiCode, Data Matrix and ArUco markers.

Fiducials can be used as references for robotic computer vision, imageprocessing, and augmented reality applications. For example, oncecaptured, the fiducials can serve as anchor points for allowing acomputer vision system to glean additional information from a capturedscene. In a specific example, available algorithms recognize an AprilTagin an image and can determine the pose and location of the tag from theimage. If the tag has been “registered” with a locale such that therelative location of the tag in the locale is known a priori, then thederived information can be used to localize other elements in the localeor determine the pose and location of the imager that captured theimage.

FIG. 1 shows a fiducial element 100 in detail. The tag holds geometricinformation in that the corner points 101-104 of the surrounding blacksquare can be identified. Based on prior knowledge of the size of thetag, a computer vision system can take in an image of the tag from agiven perspective, and the perspective can be derived therefrom. Forexample, a visible light camera 105 could capture an image of fiducialelement 100 and determine a set of values 106 that include the relativeposition of four points corresponding to corner points 101-104. Fromthese four points, a computer vision system could determine theperspective angle and distance between camera 105 and tag 100. If theposition of tag 100 in a locale were registered, then the position ofcamera 105 in the locale could also be derived using values 106.Furthermore, the tag holds identity information in that the pattern ofwhite and black squares serves as a two-dimensional bar code in which anidentification of the tag, or other information, can be stored.Returning to the example of FIG. 1 , the values 106 could include aregistered identification “TagOne” for tag 100. As such, multipleregistered tags distributed through a locale can allow a computer visionprocessing system to identify individual tags and determine the positionof an imager in the locale even if some of the tags are temporarilyoccluded or are otherwise out of the field of view of the imager.

FIG. 1 further includes a subject 110 in a set 111. As illustrated,fiducial elements 112 and 113 have been placed in set 111 to serve asreferences for facilitating the kinds of image processing techniquesmentioned above. However, as the tags have been captured along with thescene, they will need to be removed via post processing before the sceneis in final form. Furthermore, if set 111 is being used for a liveperformance, the presence of the tags could appear unprofessional and bedistracting for the audience.

SUMMARY

This disclosure includes systems and methods for generating fiducialelements using arbitrary visual features. The arbitrary visual featurescan be any visual element with sufficient texture information. They canbe natural images or objects introduced into a locale for purposes ofserving as fiducials, or natural images or objects already present inthe locale. They can also be natural images or objects naturallyassociated with or introduced to any subject. In specific embodiments ofthe invention, the visual features are natural images. In specificembodiments of the invention, the visual features include a logo orother pattern that is repeatedly presented in a given locale. Themethods of generating a fiducial element from an arbitrary visualfeature can be referred to herein as “registering” that visual feature.Once the visual feature is registered, it can be used as a fiducialelement for any of the processes described in the background above.However, if selected properly, the fiducial elements generated inaccordance with specific embodiments of the invention disclosed hereinwill not require removal in post processing or be visually obtrusive toan observer of the locale or subject on or around which the fiducialelement are located.

Registered visual features can be deployed in a given locale as afiducial element for capture by an imager operating in that locale.Locales in which the fiducial elements are deployed can include a set,playing field, race track, stage, or any other locale in which an imagerwill operate to capture data inclusive of the data embodied by thefiducial element. The locale can include a subject to be captured by theimager along with the fiducial elements. The locale can host a scenethat will play out in the locale and be captured by the imager alongwith the fiducial elements. The registered visual features can also bedeployed on a given subject as a fiducial element for capture by animager serving to follow that subject. For example, the registeredvisual features could be on the clothes of a human subject, attached tothe surface of a vehicular subject, or otherwise attached to a mobile orstationary subject. As the visual features may already be features ofthe locale or subject before they are registered, “deploying” the visualfeature does not necessarily require the physical addition of theregistered visual feature to the locale or subject so much as simplyregistering and using the visual feature during capture.

In a specific embodiment of the invention a method is disclosed. Themethod comprises aligning a geometric reference object and a visualfeature and capturing, while the geometric reference object and thevisual feature are aligned, an image of the geometric reference objectand the visual feature. A non-limiting example of a geometric referenceobject that can be used in specific embodiments of the invention is anAprilTag. A non-limiting example of a visual feature that can be used inspecific embodiments of the invention is a logo. The method alsocomprises identifying, in the image of the geometric reference objectand the visual feature, a set of at least four non-colinear featurepoints in the visual feature. The method also comprises deriving, fromthe image of the geometric reference object and the visual feature, acoordinate system using the geometric object. The method also comprisesproviding a set of measures to each of the points in the set of at leastfour non-colinear feature points using the coordinate system. Themeasures can then be saved in a memory to represent the registeredvisual feature and serve as the basis for using the registered visualfeature as a fiducial element.

In specific embodiments of the invention, a visual feature that has beenregistered using a method in accordance with the specific embodimentsdescribed in the prior paragraph can then be deployed for use as afiducial element. In a specific embodiment of the invention, a method ofusing a registered visual feature is disclosed. The method comprisesplacing the visual feature in a locale or on a subject. The method alsocomprises capturing an image of the visual feature in the locale and/oron the subject. The method also comprises deriving, from the image ofthe visual feature in the locale and using the set of measures, a poseof the visual feature in the locale or the imager in the locale.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a locale with fiducial elements inaccordance with the related art.

FIG. 2 is a flow chart of a set of methods for registering and deployingarbitrary visual features as fiducial elements in accordance withspecific embodiments of the invention.

FIG. 3 is an illustration of two potential implementations of thealigning step in the flow chart of FIG. 2 that are in accordance withspecific embodiments of the invention.

FIG. 4 is an illustration of two potential implementations of thefeature point identification step in FIG. 2 in accordance with specificembodiments of the invention.

FIG. 5 is an illustration of a potential implementation of the measureprovisioning step in FIG. 2 in accordance with specific embodiments ofthe invention.

FIG. 6 is an illustration of a locale with fiducial elements that havebeen registered in accordance with specific embodiments of theinvention.

DETAILED DESCRIPTION

Specific methods and systems associated with visual features inaccordance with the summary above are provided in this section. Themethods and systems disclosed in this section are non-limitingembodiments of the invention, are provided for explanatory purposesonly, and should not be used to constrict the full scope of theinvention.

FIG. 2 illustrates a flow chart for a set of methods that are inaccordance with specific embodiments of the invention. The flow chartincludes a first section 200 in which a visual feature is registered foruse as a fiducial element and a second section 210 in which the visualfeature is deployed for such use. The flow chart includes the example ofa visual feature in the form of a distinctively textured section of abrick wall 250 and a geometric reference object in the form of a QR codetag 260. However, the approaches disclosed herein can be used with abroad array of visual features and geometric reference objects invarious combinations.

In specific embodiments of the invention, any arbitrary visual featurehaving sufficient texture or depth information can be registered as afiducial element. The visual features can include natural images thatcan be introduced to a given locale or natural features of the localeitself. The visual features can be two-dimensional surfaces orthree-dimensional objects. The natural images can include posters orbanners with advertisements, team insignias, or company logos. Thenatural features can include distinctive surfaces such as brick wall 250or three-dimensional objects that are already in the locale such asfurniture or natural landmarks. In specific embodiments, set pieces canbe introduced to a locale specifically because they have sufficienttexture information for a given application but otherwise appear to be anatural element of the locale (e.g., a wood table with a distinctivewood grain deployed on the set of a family dining room). The amount oftexture or depth information that is sufficient for registering a visualelement will depend on the implementation and will be addressed in moredetail below.

In specific embodiments of the invention, the geometric reference objectcan take on different forms. The geometric reference objects can includefiducial elements such as AprilTags, ArUco markers, circular patternedfiducial markers, and any two-dimensional or three-dimensional fiducialelement used in the robotics, computer vision, and augmented realityfields. In general, any geometric reference object from which ann-dimensional geometry can be derived could be utilized as the geometricreference object in accordance with this disclosure where “n” is thenumber of dimensions of the visual feature being registered. Forexample, in the case of a Euclidean geometry with a two-dimensionalvisual feature, a common L-square carpenter's ruler with a known handlelength could serve as the geometric reference object. In specificembodiments, the geometric reference object could be a customizedmeasuring tool designed to stick temporarily to surfaces on a set. Thedevice could involve one or more integrated levels for aligning thegeometric object with a visual feature in accordance with specificembodiments of this disclosure. However, geometric objects in the formof AprilTags and other fiducial elements that are in common use arebeneficial because there are readily available software tools forsegmenting them from captured images and deriving location and poseinformation from the segmented information.

In specific embodiments of the invention, a grid of geometric referenceobjects will be used as opposed to a single reference object. Forexample, a grid of AprilTags. Although a single AprilTag of known sizecould be sufficient in some applications, the use of additional tagswill enable the system to screen out errors due to nonidealities of animager such as image noise or the size of the pixel receptors of animager. A grid of tags can provide an accurate reading because there arenumerous edges and corners to be detected and a potential bias, owing totag edges and corners being located in sub-pixel locations, can bescreened out in the next row or column of the grid. Furthermore,different sets of points derived from the geometric object can beselected and utilized to determine measures and or camera poses and thebest consensus can be used to define the selected points using a randomsample consensus (RANSAC) approach.

Flow chart section 200 begins with a step 201 of aligning a geometricreference object with a visual feature. This step can include placingthe geometric reference object proximate to the visual feature withoutany appreciable degree of attention placed on the specific manner inwhich the visual feature is physically aligned with the geometricreference object so long as the geometric reference object and visualfeature are aligned in at least one dimension. For example, both thegeometric reference object and the visual feature could be placed on thesame surface. This could involve placing both on the floor or adheringthe reference object to a wall or other surface on which the visualfeature was already located. In the illustrated situation of FIG. 2 , QRcode tag 260 is effectively aligned with visual element 250 in that QRcode tag 260 is taped or otherwise adhered to the wall such that boththe visual element and geometric reference object lie in the same plane.

In other embodiments, such as those in which the geometric referencesobject and/or the visual feature are three-dimensional, alignment caninvolve aligning an alignment feature of both the geometric referenceobject and the visual feature to a common plane. The plane could be aplane normal to a surface on which both elements were placed. Forexample, in the case of a three-dimensional reference object and visualfeature, the aligning step could be conducted by placing thethree-dimensional object and the visual feature proximate to each other,on a surface, and with a first alignment feature of thethree-dimensional object and a first alignment feature of the visualfeature located on a plane that is normal to the surface. The alignmentof the two elements along the common plane and the surface would therebyprovide alignment information for a three-dimensional alignment of thetwo.

Alignment features used in accordance with specific embodiments of thepresent invention can take on various forms. For example, the alignmentfeatures could be an inherent physical characteristics of an elementsuch as the center or edge of an object. As another example, thealignment features could be a visual aspect of an object such as thecenter line of a table. The alignment features could also be added to anobject such as by marking the object with a pen or tape. The alignmentfeatures should generally be selected such that they allow a user tointuitively and easily align the geometric reference object and visualfeature via visible inspection and minimal effort.

FIG. 3 provides two explicit examples 300 and 310 of the result of anexecution of step 201 in accordance with specific embodiments of thepresent invention in which the alignment is conducted manually by auser. In example 300, a set of AprilTags 301 has been placed next to acompany logo 302 on the floor. For example, the company logo could be ona banner placed on the floor next to the set of AprilTags. The companylogo 302 and set of AprilTags 301 are aligned in this image because theyare located in the same two-dimensional plane, defined by the floor ofthe locale in which image 300 was captured. In example 310, table 311and reference object 312, in the form of a cube with each facedisplaying a separate AprilTag, are aligned because they are placed onthe same surface 313 and because an alignment feature of each elementhas been placed in a common plane 314. In the illustrated case, thecenterline of table 315 and a reference point 316 on reference object312 have been placed in common plane 314. Common plane 314 is normal tosurface 313. Table 311 includes an alignment feature in the form ofcenterline 315, and it also includes a symmetry reduction feature in theform of cup 317. Centerline 315 is visible to the naked eye andphysically located on table 311 to assist a user in conducting step 201.Symmetry reduction features for arbitrary visual features, such as cup317, are important in certain embodiments as described below.

Flow chart section 200 continues with a step 202 of capturing an image.The image can be captured by an imager and include the geometricreference object and the visual feature as aligned in step 201. Theimager can be a single visible light camera such as a standard herocamera used in professional motion picture capture applications. Theimager can also include an auxiliary or witness camera. The witness andhero cameras can both be visible light cameras and can be used incombination in stereo fashion to obtain depth information such that theimage can included depth information derived from the two cameras. Anynumber of witness cameras can be used along with the hero camera. Theimager can also include a dedicated depth sensor for extracting depthinformation for the image in combination with a visible light imager forcollecting texture information. All of the components of the imager suchas a hero camera and any auxiliary witness cameras or depth sensors canbe attached to a single rig such that they have an overlapping field ofview and can all be easily directed towards the aligned visual featureand geometric reference object at the same time. In specific embodimentsof the invention, the image can consist solely of depth information,solely of texture information, or a combination of both depending uponthe imager used. The imager can capture light of any spectrum includingvisible, ultraviolet, or infrared light.

Flow chart section 200 continues with a step 203 of identifying, in theimage captured in step 202, a set of at least four non-colinear featurepoints in the visual feature. The four non-colinear feature points willbe used to define a perspective view of a plane. Four points are neededto define the plane in three-dimensional space if they are provided in atwo-dimensional image. If only three points are provided in the form ofx- and y-coordinates in an image, the three points are not sufficient todefine the plane given that the perspective will alter the relativevalue of those three coordinates without an additional known variable totack down their relationship. The plane can be the plane in which atwo-dimensional visual feature is positioned in a locale. In specificembodiments of the invention, an M×N grid of feature points will beassociated with the visual feature through the execution of step 203.The grid will include four corners and a center point.

Step 203 is illustrated by the discovery of five feature points 270 inan image of wall 250. The feature points can be found using standardcomputer vision image processing feature finding algorithms or a trainednetwork. The network can be trained to identify feature points thatmaximize detectability of those feature points from differentperspectives, distances, changes in lighting conditions, and othervariations between when the image is captured, such as in step 202, andwhen the visual feature is deployed, such as in flow chart section 210.

In specific embodiments of the invention, step 203 can be conducted by asystem that combines standard computer vision image processing featurefinding algorithms to select a large set of candidate points, and atrained network to cull the list of candidate points for purposes ofselecting the feature points that will maximize detectability and theaccurate provisioning of geometric and/or identify information from thevisual feature at a later time during deployment. The trained networkcould be part of a network used when the visual features are deployed.

The visual feature should have sufficient textual or depth informationin order for distinctive feature points to be discovered for thispurpose. The textual or depth maps should accordingly have high entropy,low repeatability, and low internal correlation, and be asymmetrical.For example, the texture map could be an asymmetric two-dimensionaltexture map with a high degree of edginess per unit area and littlerepetition. If recognition of the visual features is to rely on textureinformation the visual feature could be a two-dimensional object with anon-repeating texture map. For example, the visual feature could be anasymmetrical picture where it is not possible to draw a line of symmetrythrough the picture. This aspect of the visual feature will assure thatan imager is able to find enough feature points to precisely derive theposition of the imager without ambiguity. The feature points should alsobe stable in the sense that they do not change position as a captureproceeds. For example, a crumpled bed sheet might have many of thefeatures described above, but the feature points of the sheet would belikely to disappear or move slightly relative to one another such thatthey no longer provide sufficient reference information for a prioriregistration and later deployment.

FIG. 4 provides two examples of sets of feature points that could beextracted from a visual feature. Visual feature 400 is a two-dimensionalnatural image in the form of a framed picture of a distinctivelandscape. A set of feature points that could be derived therefrominclude sample point 401 and four other points. Visual feature 400 isparticularly useful because there is a high degree of texture variationin the image and no symmetry such that viewing the visual feature fromeither perspective 402 or 403 would not result in an ambiguous detectionof feature points. Visual feature 410 is three-dimensional table with acenter line. However, this would not be ideal for an execution of step203 because of the symmetry of visual feature 410 along that centerline. If an imager were to detect visual feature 410 from perspective412 it would not be able to distinguish if it was viewing the visualfeature from perspective 413. As a result, a symmetry reduction element,in this case a coffee cup, has been added to the table in order for asufficient set of feature points, such as feature point 411, to bederived.

The system used to identify the feature points in step 203 could bedesigned to alert a user that insufficient feature points are availablewith any given visual feature such that a different visual featureshould be selected. For example, if the texture map of wall 250 did nothave enough variation (e.g., it was a perfect uniform brick wall withnear-identical bricks) then the system could inform that user that thewall would not be able to be registered as a visual feature for use as afiducial element. If recognition of the visual feature was to rely ondepth information, the visual feature could be a three-dimensionalobject with a nonrecurrent depth map. For example, the visual featurecould be an asymmetrical piece of furniture where it would not bepossible to draw a plane of symmetry through the object. As statedpreviously, users could be provided with the option to introducesymmetry reducing elements, such as the cup 317 on table 311 in order toallow a set piece to serve as a registered visual feature. The systemcould reject potential visual features based on a set thresholdrequirement for a degree of texture variation and could also beaugmented with information concerning a texture map of the locale ingeneral such that visual features with texture or depth maps that werecorrelated too closely with other portions of the locale would berejected.

Flow chart section 200 continues with a step 204 of deriving from theimage or the geometric reference object and the visual feature, acoordinate system using the geometric object. This step involves using aprior information regarding the geometric object to generate an ordinalreference frame from the image. In one example, the geometric referenceobject could be a physical ruler with standard units of distancedemarked thereon such that image processing conducted on the image couldidentify unit steps for the coordinate system. In another example, thegeometric reference object could have specific features such as itslength or width that were known a priori from which unit steps for thecoordinate system could be derived. Furthermore, the axes of thecoordinate system could be derived either randomly, or they could bebased on the position of certain features of the geometric object. Forexample, if the reference geometric object were two-dimensional andincluded a right angle, the right angle could be used to define the xand y axes of the coordinate system. In the case of a geometric objectin the form of a standard fiducial such as an AprilTag, the coordinatesystem could be derived using commonly available algorithms fordetermining the pose or position of such fiducials and then basing thecoordinate system off that pose. A geometric reference object with aknown physical size bounded by four detectible points in the image couldprovide the basis for a three-dimensional coordinate system.

Flow chart 200 continues with a step 207 of providing a set of measuresto each of the points in the set of at least four feature pointsidentified in step 203 using the coordinate system derived in step 204.The measures could be relative Euclidean measures. The measures could bedistances between the four points. There are numerous ways by which themeasures could be provided. For example, the measures could be assignedto each of the four points and include the distance to each of the otherpoints in the set. Alternatively, the distances to at least two otherpoints in the set could be provided to each point. Alternatively, adistance to a common point in the set could be assigned to each pointwith the common point being assigned a distance of zero. Regardless ofhow the measures are assigned as long as the system is able to derivethe expected apparent distance between each point in an image taken fromany perspective in which all the points are present will suffice forpurposes of conducting step 207 with a two-dimensional visual feature.

FIG. 5 provides illustrates a specific implementation of step 207. InFIG. 5 an image has been captured which includes a set of geometricreference objects 500 aligned with a visual feature 501. As shown,feature points such as feature point 502 have already been identified invisual feature 501 and a set of axes 503 and 504 have already beenderived from the geometric reference objects 500. The axes are ordinalwith unit sizes derived from a priori available information regardingthe geometric reference objects 500 or information encoded in thereference images. In the illustrated case, the distance from the originof the coordinate frame is them determined for each of the points andmeasures, such as measures 510 are derived therefrom in order toregister the visual feature 501 for use as a fiducial element.

Flow chart section 200 continues with an optional step of hashing dataassociated with the measures 206 and a step of generating a feature IDfor the visual feature 205. This step is optional in that it is onlyrequired if the visual feature is being registered for an application inwhich it needs to be distinguished from other visual features in asingle locale. Furthermore, there are other methods of encoding the IDof a visual feature in a scene such that this is not a required stepassociated with every embodiment of the invention. In a specificembodiment of the invention, the distances between the feature pointsare hashed and used as a distinctive ID for the registered visualfeature such that they can be recalled and the tag can be identified ata later time. The hashing step can also be optional in that the ID canbe derived from the distances or other features of the measures orimage.

In specific embodiments of the invention, the ID could be derived from apriori knowledge of the planned deployment location of the visualfeature. For example, in certain applications, multiple identicalinstantiations of a visual feature may be registered for use as fiducialelements. In particular, a team insignia or company logo may be placedin numerous locations in a given locale. In these embodiments, the ID ofthe visual feature could be derived from the proximity of the feature toanother easily detectable visual feature in the locale. In the case ofsports arenas with fixed imager positions, the ID could also be detectedbased on the approximate pose of the camera at the time which the visualfeatures was captured by the imager. In specific embodiments, variousversions of a company logo, or different brand or company logos, can beused to provide a unique signature for identification informationdirectly. Furthermore, in certain applications, specific IDs for visualelements are not needed as the system can instead register the specificlocation of visual elements relative to each other and solve for theidentification of specific visual elements when enough of them aredetected by a single imager.

Flow chart section 200 also includes an optional step 208 of training anetwork to detect visual element 250. The network can be a trainabledirected graph such as an artificial neural network (ANN). The networkcan be trained to identify the feature points of the visual element froman image of the locale or object in which the visual element is located.The training procedure can include the generation of training data withvarious lighting conditions, perspectives, occlusions, and transforms.The generation of training data can also include applying randomhomographies to a planar visual element. In specific embodiments,feature points, such as feature points 270 will be selected such thatthey can be readily detected using standard computer vision featuredetection routines. However, a network can be trained to detect a widearray of feature points under a broad array of distortions in terms ofperspective, lighting, and other variations that would not generally bedetectible using traditional methods. In addition, and as alluded topreviously, the design and training of the network could be conducted intandem with the design and optional training of the system used toselect feature points in step 203 such that the feature points used toregister the visual item were feature points that were optimized fordetection by the network under various conditions.

A network can recognize an image itself and visual elements in an imageand provide the location of the visual element in various ways. In aspecific embodiment, the network could be designed to detect N objects.The input of the network could be an encoding of the image and an outputcould be a “1” at a specific location in the vector if the object waspresent and all else zero. The output could also provide location and/orpose information. For example, the output could be an N×m matrixidentifying the precise location and pose of an object where m could besix units in size representing x-, y-, and z-coordinates as well aspitch, yaw, and roll. The output could also be the x- any y-coordinatesof at least four points with known relative measures on the object, and,so long as the pose of the image was known, the precise location andpose of the visual element could be derived therefrom. In situationswhere the visual elements were planar, the output could also be an N×mmatrix where m could be eight units and contain homography values forthe visual element with the last homograph value normalized to 1. Thehomoraphy values could be the values of a 3×3 homograph matrix for thevisual element.

Flow chart section 210 includes various steps associated with thedeployment of a visual feature that was registered to serve as afiducial element in flow chart section 200. Flow chart section 210commences with a step 211 of placing the visual feature. The visualfeature can be placed in a locale or on a subject that will be capturedby an imager along with the fiducial element. As mentioned previously,this step can be conducted prior to registering the visual element ifthe visual element is an inherent aspect of the locale or subject, suchas a textured wall, a car's hood ornament, or a painting mounted on thewall of a room.

Flow chart section 210 continues with a step 212 of capturing an imageof the subject or in the locale. The step can be conducted using thesame imager used in step 202. Using the same imager can make repeatdetection of the feature points somewhat easier but is not an essentialstep as even the same imager can have its characteristics altered overtime. Regardless, the captured image can then be analyzed by acomputerized system, either using traditional computer vision algorithmsor trained networks. The analysis can include the execution of step 213in which the pose of the visual feature is derived, from the image ofthe visual feature in the locale, using the set of measures registeredin flow chart section 200. This step can involve a network trained instep 208 identifying the feature points or deriving the pose from thefeature points, or both. The pose of the visual feature can then be usedfor any of the approaches mentioned above with reference to traditionalfiducial elements with encoded geometric information. The pose can befurther refined using photogrammetry in which an initial pose is used toalign a known model of a captured object with an image of the objectusing an optimization algorithm.

Flow chart section 210 additionally includes an optional step 214 ofderiving a unique identification of the visual feature. The ID can bethe ID registered in step 205. As with the generation of the ID, thedetection of the ID can use the measures, or it can use an alterativesystem such as by intuiting the ID from the relative location or othervisual features, a priori information regarding the camera pose relativeto the locale, or other computer vision techniques which were used todetermine the ID in step 205.

Using the approaches disclosed herein, visually unobtrusive fiducialscan be utilized in a locale such that they do not require postprocessing removal and do not present a distraction to an oberverviewing the locale at the same time as a capture with the fiducials isconducted. FIG. 6 illustrates a locale 600 with registered visualfeatures 601, 602, and 603. As seen, these visual features can benaturally occurring elements of the scene instead of large posters withdistracting black and white patterns. However, an imager 604, can stillderive geometric information 605 from these visual features, such asvisual feature 601, in the same way that information was derived fromAprilTag 100 in FIG. 1 because the measures, and potential descriptionsof the feature points, associated with visual feature 601 have beenregistered and stored for later use as an encoded fiducial element.

While the specification has been described in detail with respect tospecific embodiments of the invention, it will be appreciated that thoseskilled in the art, upon attaining an understanding of the foregoing,may readily conceive of alterations to, variations of, and equivalentsto these embodiments. While the example of a visible light camera wasused throughout this disclosure to describe how a frame is captured, anysensor can function in its place to capture a frame including depthsensors without any visible light capture in accordance with specificembodiments of the invention. Any of the method steps discussed above,with the exception of the aligning and placing steps which involvephysical manipulations of the visual features or geometric referenceobject, can be conducted by a processor operating with acomputer-readable non-transitory medium storing instructions for thosemethod steps. The computer-readable medium may be memory within apersonal user device or a network accessible memory. Modifications andvariations to the present invention may be practiced by those skilled inthe art, without departing from the scope of the present invention,which is more particularly set forth in the appended claims.

What is claimed is:
 1. A computer-implemented method comprising:registering a visual feature as a fiducial element, the registeringincluding: aligning a reference object and a visual feature; capturingan image of the reference object and the visual feature; identifying, inthe image, a set of feature points in the visual feature; deriving, fromthe image, a coordinate system using the reference object; and providinga set of measures to each feature point of the set of feature pointsusing the coordinate system; and placing the registered visual featurein a locale.
 2. The computer-implemented method of claim 1, furthercomprising: capturing an image of the visual feature in the locale usingan imager; and deriving, from the image of the visual feature in thelocale and using the set of measures, a pose of the visual feature inthe locale or the imager in the locale.
 3. The computer-implementedmethod of claim 2, further comprising: generating, using data associatedwith the set of feature points, a unique identifier for the visualfeature; and determining, from the image of the visual feature in thelocale, the unique identifier for the visual feature.
 4. Thecomputer-implemented method of claim 1, wherein: the reference object isa two-dimensional fiducial tag; the visual feature is two-dimensional;and the aligning the reference object and the visual feature isconducted by placing the two-dimensional fiducial tag and the visualfeature proximate to each other on a surface.
 5. Thecomputer-implemented method of claim 4, wherein capturing the image isconducted using a single visible light image capture device.
 6. Thecomputer-implemented method of claim 1, further comprising: identifying,in the image of the reference object and the visual feature, analignment feature of the reference object and an alignment feature ofthe visual feature; the reference object is a three-dimensional object;the visual feature is three-dimensional; and the aligning the referenceobject and the visual feature is conducted by placing thethree-dimensional object and the visual feature: (i) proximate to eachother; (ii) on a surface; and (iii) with the alignment feature of thethree-dimensional object and the alignment feature of the visual featurelocated on a plane that is normal to the surface.
 7. Thecomputer-implemented method of claim 6, wherein: the capturing the imageis conducted using a hero visible light camera and a witness visiblelight camera; and the image includes depth information derived from thehero visible light camera and the witness visible light camera.
 8. Thecomputer-implemented method of claim 1, wherein the visual feature is atwo-dimensional picture with a nonrecurrent texture map.
 9. Thecomputer-implemented method of claim 1, wherein the visual feature is atwo-dimensional asymmetrical picture.
 10. The computer-implementedmethod of claim 1, the reference object is a regular two-dimensionalarray of two-dimensional fiducial tags; the image consists of a set ofpixels spaced according to a set of pixel receptors of an imager used tocapture the image; and the set of feature points are sub-pixel locationsrelative to the set of pixels.
 11. The computer-implemented method ofclaim 1, hashing data associated with the set of feature points togenerate a unique identifier for the visual feature; placing the visualfeature in a locale; capturing an image of the visual feature in thelocale; and identifying, from the image of the visual feature in thelocale, the unique identifier for the visual feature.
 12. Thecomputer-implemented method of claim 11, wherein the data associatedwith the set of feature points are Euclidean measures.
 13. Thecomputer-implemented method of claim 1, wherein the visual feature is afirst visual feature and the set of feature points is a first set offeature points, the method further comprising: registering a secondvisual feature using the reference object to provide a second set ofEuclidean measures to a second set of feature points using thecoordinate system; hashing data associated with the first set of featurepoints to generate a first unique identifier for the first visualfeature; hashing data associated with the second set of feature pointsto generate a second unique identifier for the second visual feature;placing the first visual feature and the second visual feature in alocale; capturing an image of both the first visual feature and thesecond visual feature in the locale; and identifying, from the image ofboth the first visual feature and the second visual feature in thelocale, the first unique identifier for the visual feature and thesecond unique identifier for the second visual feature.
 14. Thecomputer-implemented method of claim 1, wherein the identifying isconducted using an automatic feature detector and a network trained toselect points for maximum detectability.
 15. The computer-implementedmethod of claim 1, wherein the set of measures are distances to at leasttwo other points in the set.
 16. The computer-implemented method ofclaim 1, wherein the set of measures are distances to a common point inthe set.
 17. The computer-implemented method of claim 1, furthercomprising: training a network to segment and identify the visualfeature in any image; placing the visual feature in a locale; capturingan image of the visual feature in the locale; and deriving, from theimage of the visual feature in the locale, a pose of the visual featurein the locale using the set of measures and the network.
 18. Anon-transitory computer-readable medium storing instructions forexecuting a method comprising: registering a visual feature as afiducial element, the registering including: aligning a reference objectand a visual feature; capturing an image of the reference object and thevisual feature; identifying, in the image, a set of feature points inthe visual feature; deriving, from the image, a coordinate system usingthe reference object; and providing a set measures to each feature pointof the set of feature points in the visual feature using the coordinatesystem; and placing the registered visual feature in a locale.